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Introduction 


/ 


•  Technology  Readiness  Levels  are  a  scale  that 
describes  the  maturity  of  a  technology  with 
respect  to  a  particular  use 

-  Scale  from  1  (least  mature)  to  9  (most  mature) 

-  Heuristics: 

-  TRL  1  =  “an  idea” 

-  TRL  4  =  “a  software  mock-up” 

-  TRL  6  =  “a  prototype  that  has  completed  beta  testing” 

-  TRL  7  =  “ready  for  operational  testing,  technical  work  complete” 

-  TRL  9  =  “fielded  and  used  as  intended” 


To  establish  a  TRL  valuey  an  understanding  of  BOTH  the  accomplishments 
and  the  intended  use  of  the  technology  is  required 
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Technology  Readiness  Assessments 

are  Narrow  in  Scope 


•  Primary  Purpose:  to  help  management  make  decisions  concerning  the 
development  and  transition  of  technology. 

•  Uses  of  Technology  Readiness  Levels  (TRLs): 

-  Provides  a  common  understanding  of  technology  status  (maturity) 

-  Conveys  what  has  been  accomplished  (demonstrated)  with  the  technology 

-  Used  as  a  factor  in  technical  risk  management 

-  Used  to  make  decisions  concerning  technology  funding 

-  Used  to  make  decisions  concerning  transition  of  technology 

-  Used  to  scope  acquisition  programs  and  their  requirements 

-  Used  as  a  basis  for  certification  under  statute 

•  Other  risk-related  technical  measures  are  also  important: 

-  Performance  envelope  of  the  technology  -  where  does  the  technology  break? 

-  Difficulty  of  work  to  be  done  -  can  the  technology  be  matured? 

-  Systems  Design  -  is  the  design  good? 

-  Systems  engineering  and  systems  integration  -  are  the  system  design  objectives 
met? 

-  Appropriateness  of  the  technology  -  is  it  the  best  choice? 


The  TRL  value  does  not  indicate  that  the  technology  is  right  for  the  job  or  that 
application  of  the  technology  will  result  in  successful  development  of  the  system 
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Technology  Readiness  Assessments 

(TRAs): 

Historical  Perspective 

•  “Program  managers’  ability  to  reject  immature  technologies  is  hampered  by 

(1)  untradable  requirements  that  force  acceptance  of  technologies  despite  immaturity, 
and 

(2)  reliance  on  tools  that  fail  to  alert  managers  of  high  risks  that  would  prompt 
rejection.” 

GAO/NSI AD-99-1 62 

•  “Identify  each  case  in  which  a  major  defense  acquisition  program  entered  system 
development  and  demonstration  ...  into  which  key  technology  has  been  incorporated 
that  does  not  meet  the  technology  maturity  requirement ...  and  provide  justification  for 
why  such  key  technology  was  incorporated  and  identify  any  determination  of 
technological  maturity  with  which  the  Deputy  Under  Secretary  of  Defense  for  Science 
and  Technology  did  not  concur  and  explain  how  the  issue  has  been  resolved.” 

National  Defense  Authorization  Act  for  Fiscal  Year  2002 

•  “The  management  and  mitigation  of  technology  risk,  which  allows  less  costly  and  less 
time-consuming  systems  development,  is  a  crucial  part  of  overall  program  management 
and  is  especially  relevant  to  meeting  cost  and  schedule  goals.  Objective  assessment  of 
technology  maturity  and  risk  shall  be  a  routine  aspect  of  DoD  acquisition.” 

DoDI  5000.2,  paragraph  3. 7. 2. 2 

Aim:  To  stop  launching  acquisition  programs  before  technologies  are  mature! 
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Defense  Acquisition  Framework 


Prerequisites  for  new  programs: 

•  Validated  operational  requirement 


User  Needs 


Technology  Opportunities  &  Resources 


Technology/acquisition  readiness 
Identified  funding  resources 


TRL6 


TRL  7 


(Pr< 

Initj 


gram 

ition) 


IOC 


FOC 


A  N 


Materi 
Solution 
Analysis 


Materiel 
Development 
Decision 


— ’ Tec  h  no  lo  gy  — 
Development 

Engineering  and 
Manufacturing 
Development 

Production  & 
Deployment 

Operations  & 
Support 

Post-  ^^^Post- 
V  PDR  A  V  CDR  A 

LRI P/IOT&E  /S  Decision 

Nr  Review 

Pre-Systems  Acquisition^X^ 


7\ 


Systems  Acquisition 


Sustainment 


Technology  Readiness  Levels  are  one  indicator  of  readiness  to 
commence  succeeding  stages  of  acquisition 
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Changes  to  Statute 


10  USC  §2366b  states 

Major  defense  acquisition  programs:  certification  required  before  Milestone  B 
or  Key  Decision  Point  B  approval: 

(a)  CERTIFICATION.  A  major  defense  acquisition  program  may  not  receive 

Milestone  B  approval,  or  Key  Decision  Point  B  approval  in  the  case  of  a  space 
program,  until  the  milestone  decision  authority  certifies  that  [the  MDA]- 


(2)  has  received  the  results  of  the  preliminary  design  review  and  conducted  a  formal  post-preliminary  design 
review  assessment,  and  certify  on  the  basis  of  such  assessment  that  the  program  demonstrates  a 
high  likelihood  of  accomplishing  its  intended  mission 


(3)(D)  the  technology  in  the  program  has  been  demonstrated  in  a  relevant  environment  as 

determined  by  the  Milestone  Decision  Authority  on  the  basis  of  an  independent  review  and 
assessment  by  the  Director  of  Defense  Research  and  Engineering; 

(3)(E)  the  program  complies  with  all  relevant  policies,  regulations  and  directives  of  the  Department  of 
Defense 


In  addition,  P.L.  111-23  (WSARA)  amended  Title  10  (139  a(c)(2)  of  title  10)  to 
require  DDR&E  to  directly  report  TRA  findings  annually  to  the  Congress 


All  technologies  in  large  programs  must  be  TRL  6  or  better  as  determined  by 
DDR&E  unless  waived  by  the  MDA  for  national  security  reasons 
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Department  of  Defense  Hardware  TRLs 


Basic  principles  observed  and  reported 

Technology  concept  and/or  application 
formulated 

Analytical  and  experimental  critical  function 
and/or  characteristic  proof  of  concept 

Component  and/or  breadboard  validation  in 
a  laboratory  environment 

Component  and/or  breadboard  validation  in 
a  relevant  environment 

System/subsystem  model  or  prototype 
demonstration  in  a  relevant  environment 

System  prototype  demonstration  in  an 
operational  environment 

Actual  system  completed  and  qualified 
through  test  and  demonstration 

Actual  system  proven  through  successful 
mission  operations 
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Department  of  Defense  Software  TRLs 


Basic  principles  observed  and  reported. 
Technology  concept  and/or  application  formulated. 


Analytical  and  experimental  critical  function  and/or 
characteristic  proof  of  concept 

Module  and/or  subsystem  validation  in  a 
laboratory  environment,  i.e.  software  prototype 
development  environment 

Module  and/or  subsystem  validation  in  a  relevant 
environment 

Module  and/or  subsystem  validation  in  a  relevant 
end-to-end  environment 

System  prototype  demonstration  in  an  operational 
high  fidelity  environment 

Actual  system  completed  and  mission  qualified 
through  test  and  demonstration  in  an  operational 
environment 


Actual  system  proven  through  successful  mission 
proven  operational  capabilities 
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Definition  and  Purpose: 
Technology  Readiness  Assessment  (TRA) 


•  DoD  Requires  all  programs  to  conduct  a  TRA  before 
commencing  Engineering,  Manufacturing  Design 
Development  (that  is,  at  Milestone  B) 

TRA  Definition: 

Systematic,  metrics-based  process  and  accompanying  report 

•  Assesses  the  maturity  of  Critical  Technology  Elements 
(CTEs)  used  in  systems 

•  Uses  Technology  Readiness  Levels  (TRLs)  as  the  metric 

-  Adequate  maturity  at  MS  B  (TRL  6  or  greater)  is  largely 
based  on  experience  with  prototypes  or  previous  usage  in  a 
relevant  environment 

•  Report  includes 

-  how  the  CTEs  are  identified, 

-  why  CTEs  are  important  to  the  program,  and 

-  an  independent  (from  the  program)  assessment  of  their 
maturity 


DoD  uses  the  TRA  as  the  basis  for  the  certification  to  Congress 
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Critical  Technology  Elements 


A  technology  element  is  “critical”  if  the  system  being  acquired  depends 
on  this  technology  element  to  meet  its  operational  requirements  (within 
acceptable  cost  and  schedule  limitations)  and  if  the  technology  element  or 
its  application  is  either  new  or  novel  or  in  an  area  that  poses 
technological  risk  during  detailed  design  or  demonstration. 


•  Software  CTEs  can  be  new  developments  (new  or  novel)  or 
COTs/GOTs  (new  or  novel  applications) 

•  Software  CTEs  -  and  programs  -  typically  stumble  over  the 
evaluation  of  the  environment  in  which  it  will  be  employed 
-  although  COTs  may  be  commonly  used  commercially,  the 
DoD  operational  and  technical  environment  is  often 
different  from  the  commercial  environment  (think,  e.g.,  the 
technical  security  environment 

Architectures/Design  and  test  data  usually  need  to  identify  and 

evaluate  software  CTEs  with  confidence 
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TRA  Process  Overview 


Program  Responsibility  - 

coordinated  with  DDR&E  and  S&T 
Executive 

Independent  review  team 
appointed  by  S&T  Executive 
(membership  coordinated  with 
DDR&E)  selects  the  CTEs 


S&T  Executive 
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Appoints  independent  review  team 
to  assess;  Program  Funds 
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Component  (AF,  Army,  Navy, 
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Director,  Defense  Research  & 

Engineering/Research 

Directorate 
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TRA  Execution 


1 .  Independent  Review  Team  identifies  the  CTEs  by  looking  across  the  established 
program  Work  Breakdown  Structure  (WBS)  or  Systems  Architecture  for 

technology  components  essential  to  the  system  and  are  either  new  or  novel,  are  being 
applied  in  a  new  or  novel  way,  or  are  otherwise  important  to  the  program 

2.  Data  concerning  the  performance  of  the  CTEs  are  collected  and  presented  to 
reviewers  independent  from  the  program  and  expert  in  the  technologies 

3.  Independent  reviewers  assess  maturity  of  CTEs  against  established  TRL  metrics 

4.  Assessment  is  approved  by  the  Service  or  Agency  Science  &  Technology  Executive 
and  forwarded  to  the  Component  Acquisition  Executive  (CAE)  or  Agency  Head, 
who  then  transmits  it  to  the  Research  Directorate,  DDR&E 

5.  Director  of  Research  concurs  with  the  TRA,  concurs  with  reservation,  or  does  not 
concur 

•  Director  of  Research  may  elect  to  conduct  his  own  assessment 

•  If  Director  of  Research  does  not  concur,  TRA  is  returned  to  the  Service  or  Agency 
for  changes  or  election  to  conduct  another  independent  TRA 

•  In  all  cases,  DDR&E/Research  forwards  recommendations  to  the  Milestone 
Decision  Authority  (MDA)  as  input  to  the  acquisition  decision  process 

If  all  technologies  are  TRL  6 ,  then  DDR&E  recommends  certification 
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Other  Outcomes  are  possible 


/ 


When  the  Technologies  are  not  all  TRL  6: 

-  The  program  is  re-structured  to  use  only  mature 
technologies 

-  The  program  start  is  delayed  to  mature  the 
technologies 

-  The  requirements  for  the  program  are  modified 

-  DoD  may  grant  a  waiver  for  national  security  reasons 

-  The  program  is  never  initiated  and  a  different  solution  is 
sought 


DoD  Uses  TRLs  to  help  scope  programs  and  their  requirements 
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Current  Emphasis 


•  DoD  has  recently  updated  the  TRA  Policy 

•  Past  experiences  suggest  that  for  good  outcomes: 

-  Sound  Systems  Engineering/Design  used  as  a  basis  for  CTE 
identification 

-  An  early  start  to  understanding  the  technologies  and  the 
requirements 

-  Prototyping  before  program  start 

-  Technology  maturity  guiding  requirements  development 

-  Independence  and  Expertise  of  the  assessors 

-  Selection  of  the  CTEs  to  be  assessed 

-  Assessments  folded  into  risk  management 

-  Additional  technical  assessments  (design  reviews, 
manufacturability,  e.g.) 

-  Frequent  explanations  of  TRAs  (training)  across  the  DoD 
community 

Current  update  incorporates  many  lessons  and  reflect  certification  requirements 
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TRL  Definitions 


•  BREADBOARD:  Integrated  components  that  provide  a  representation  of  a  system/subsystem  and 
which  can  be  used  to  determine  concept  feasibility  and  to  develop  technical  data.  Typically 
configured  for  laboratory  use  to  demonstrate  the  technical  principles  of  immediate  interest.  May 
resemble  final  system/subsystem  in  function  only. 

•  “HIGH  FIDELITY”:  Addresses  form,  fit  and  function.  High  fidelity  laboratory  environment  would 
involve  testing  with  equipment  that  can  simulate  and  validate  all  system  specifications  within  a 
laboratory  setting. 

•  “LOW  FIDELITY”:  A  representative  of  the  component  or  system  that  has  limited  ability  to  provide 
anything  but  first  order  information  about  the  end  product.  Low  fidelity  assessments  are  used  to 
provide  trend  analysis. 

•  MODEL:  A  reduced  scale,  functional  form  of  a  system,  near  or  at  operational  specification.  Models 
will  be  sufficiently  hardened  to  allow  demonstration  of  the  technical  and  operational  capabilities 
required  of  the  final  system. 

•  OPERATIONAL  ENVIRONMENT :  Environment  that  addresses  all  of  the  operational  requirements  and 
specifications  required  of  the  final  system  to  include  platform/packaging. 

•  PROTOTYPE:  The  first  early  representation  of  the  system  which  offers  the  expected  functionality  and 
performance  expected  of  the  final  implementation.  Prototypes  will  be  sufficiently  hardened  to  allow 
demonstration  of  the  technical  and  operational  capabilities  required  of  the  final  system. 

•  RELEVANT  ENVIRONMENT :  Testing  environment  that  simulates  the  key  aspects  of  the  operational 
environment. 

•  SIMULATED  OPERATIONAL  ENVIRONMENTAL:  Environment  that  can  simulate  all  of  the  operational 
requirements  and  specifications  required  of  the  final  system  or  a  simulated  environment  that  allows 
for  testing  of  a  virtual  prototype  to  determine  whether  it  meets  the  operational  requirements  and 
specifications  of  the  final  system. 
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